Raw Strings

Observant readers may note that the regular expression was specified slightly differently: r'fox'. The r character that precedes the string stands for “raw” (no, it does not stand for “regex”).

The difference between a raw string and a regular string is simply that raw strings do not interpret the \ character as an escape character. This means that, for example, it is not possible to escape a quote character to avoid concluding your string.

However, raw strings are particularly useful for regular expressions because the regular expression engine itself needs the \ character for its own escaping at times. Therefore, using raw strings for regular expressions is very common and very useful. In fact, it is so common that some syntax-highlighting engines will actually provide regular-expression syntax highlighting within raw strings.

Match Objects

Match objects have several methods to tell you things about the match. The group method is arguably the most important. It returns a string with the text of the match, as shown here:

>>> match = re.search(r'fox', 'The quick brown fox jumped...')
>>> match.group()
'fox'

You may be curious why this method is named group. This is because regular expressions can be split into multiple subgroups that call out just a subsection of the match. You learn more about this shortly.

Match objects have several other methods. The start method provides the index in the original string where the match began, and the end method provides the index in the original string where the match ended.

The groups and groupdict methods are used to call out subsections of the regular expression. You learn more about these methods later, during a discussion about regular expressions with backreferences.

Finally, the re attribute contains the regular expression used in the match, the string attribute ­contains the string used as the haystack, and the pos attribute is set to the position in the string where the search began.

Finding More Than One Match

A limitation of re.search is that it only returns at most one match, in the form of a match object (discussed in more detail shortly). If multiple matches exist within the string, re.search will only return the first one. Often, this is exactly what you want. However, sometimes you want multiple matches if multiple matches exist.

The re module provides two functions for this purpose: findall and finditer. Both of these methods return all non-overlapping matches, including empty matches. The re.findall method returns a list, and re.finditer returns a generator.

However, there is a key difference here. These methods do not actually return a match object. Instead, they return simply the match itself, either as a string or a tuple, depending on the content of the regular expression.

Consider an example of findall:

>>> import re
>>> re.findall(r'o', 'The quick brown fox jumped...')
['o', 'o']

In this case, it returns a list with two o characters, because the o character appears twice in the string.

Character Classes

Character classes enable you to specify that a single character should match one of a set of possible characters, rather than just a single character. You can denote a character class by using square brackets and listing the possible characters within the brackets.

For example, consider a regular expression that should match either Python or python: [Pp]ython.

What is happening here? The first token in the regular expression is actually a character class with two options: P and p. Either character will match, but nothing else. The remaining five characters are just literal characters.

What does the following regular expression match?

>>> re.search(r'[Pp]ython', 'Python 3')
<_sre.SRE_Match object; span=(0, 6), match='Python'>
>>> re.search(r'[Pp]ython', 'python 3')
<_sre.SRE_Match object; span=(0, 6), match='python'>

This regular expression matches the word Python in the string Python 3 and the word python in the string python 3. It does not make the entire word case-insensitive, though. It does not match the word in all caps, for example.

>>> re.search(r'[Pp]ython', 'PYTHON 3')
>>>

Another use for this kind of character class is for words with multiple spellings. The regular expression gr[ae]y will match either gray or grey, allowing you to quickly identify and extract either spelling.

>>> re.search(r'gr[ae]y', 'gray')
<_sre.SRE_Match object; span=(0, 4), match='gray'>

It is also worth noting that character classes like this match one and exactly one character.

>>> re.search(r'gr[ae]y', 'graey')
>>>

Here, the regular expression engine successfully matches the literal g, then the literal r. Next, the engine is given the character class [ae], and matches it against the a. Now, the character class has been matched, and the engine moves on. The next character in the regular expression is a y, but the next character in the string is an e. This is not a match, so the regular expression parser moves on, starting over and looking for a starting g. When it gets to the end of the string and fails to find one, it returns None.

>>> re.search(r'[a-zA-Z]', 'x')
<_sre.SRE_Match object; span=(0, 1), match='x'>
>>> re.search(r'[a-zA-Z]', 'B')
<_sre.SRE_Match object; span=(0, 1), match='B'>

>>> re.search(r'[ˆa-z]', '4')
<_sre.SRE_Match object; span=(0, 1), match='4'>
>>> re.search(r'[ˆa-z]', '#')
<_sre.SRE_Match object; span=(0, 1), match='#'>
>>> re.search(r'[ˆa-z]', 'X')
<_sre.SRE_Match object; span=(0, 1), match='X'>
>>> re.search(r'[ˆa-z]', 'd')
>>>

>>> re.search(r'n[ˆe]', 'final')
<_sre.SRE_Match object; span=(2, 4), match='na'>

>>> re.search(r'n[ˆe]', 'jasmine')
>>>

>>> re.search(r'n[ˆe]', 'Python')
>>>

>>> re.search(r'\bcorn\b', 'corn')
<_sre.SRE_Match object; span=(0, 4), match='corn'>
>>> re.search(r'\bcorn\b', 'corner')
>>>

>>> re.search(r'\w', 'Python 3')
<_sre.SRE_Match object; span=(0, 1), match='P'>

>>> re.findall(r'\w', 'Python 3')
['P', 'y', 't', 'h', 'o', 'n', '3']

>>> re.search(r'corn\B', 'corner')
<_sre.SRE_Match object; span=(0, 4), match='corn'>
>>> re.search(r'corn\B', 'corn')
>>>

>>> re.search(r'ˆPython', 'This code is in Python.')
>>> re.search(r'ˆPython', 'Python 3')
<_sre.SRE_Match object; span=(0, 6), match='Python'>

>>> re.search(r'fox$', 'The quick brown fox jumped over the lazy dogs.')
>>> re.search(r'fox$', 'The quick brown fox')
<_sre.SRE_Match object; span=(16, 19), match='fox'>

>>> re.search(r'p.th.n', 'python 3')
<_sre.SRE_Match object; span=(0, 6), match='python'>
>>> re.search(r'p..hon', 'python 3')
<_sre.SRE_Match object; span=(0, 6), match='python'>

Optional Characters

Thus far, all of the regular expressions you have seen have involved a 1:1 correlation between ­characters in the regular expression itself and characters in the string being searched.

Sometimes, however, a character may be optional. Consider again the example of a word with more than one correct spelling, but this time, the inclusion of a letter is what separates the two spellings, such as “color” and “colour,” or “honor” and “honour.”

You can specify a character, character class, or other atomic unit within a regular expression as optional by using the ? character, which means that the regular expression engine will expect the token to occur either zero times or once.

For example, you can match the word “honor” with its British spelling “honour” by using the regular expression honou?r.

>>> import re
>>> re.search(r'honou?r', 'He served with honor and distinction.')
<_sre.SRE_Match object; span=(15, 20), match='honor'>
>>> re.search(r'honou?r', 'He served with honour and distinction.')
<_sre.SRE_Match object; span=(15, 21), match='honour'>

In both cases, the regular expression contains four literal characters, hono. These match the hono in both honor and honour. The next thing that the regular expression hits is an optional u. In the first case, the u is absent, but this is okay because the regular expression marks it as optional. In the second case, the u is present, which is also okay. In both cases, the regular expression then seeks a literal r character, which it finds, therefore completing the match.

Repetition

Thus far, you have learned only about characters (or character classes) that occur once and exactly once, or that are entirely optional (occurring zero times or once). However, sometimes you need the same character or character class to repeat.

You may expect a character class to recur a set number of consecutive times, such as in a phone number. American phone numbers comprise the country code 1 (often omitted), an area code, which is three digits, then the seven-digit phone number, with the third and fourth digit of the latter separated by a hyphen, period, or similar.

You can designate that a token must repeat a given number of times with {N}, where the N character corresponds to the number of times the token should repeat.

The following uses a regular expression to identify a seven-digit, local phone number (ignore the country code and area code for the moment): [\d]{3}-[\d]{4}.

>>> re.search(r'[\d]{3}-[\d]{4}', '867-5309 / Jenny')
<_sre.SRE_Match object; span=(0, 8), match='867-5309'>

In this case, the regular expression engine starts by looking for three consecutive digits. It finds them (867), and then moves on to the literal hyphen character. Because this hyphen character is not within a character class, it carries no special meaning and simply matches the literal hyphen. The regular expression then finds the final four consecutive digits (5309) and returns the match.

>>> re.search(r'[\d]{3,4}', '0421')
<_sre.SRE_Match object; span=(0, 4), match='0421'>
>>> re.search(r'[\d]{3,4}', '615')
<_sre.SRE_Match object; span=(0, 3), match='615'>

>>> re.search(r'[\d]{3,4}?', '0421')
<_sre.SRE_Match object; span=(0, 3), match='042'>

>>> re.search(r'[\d]{1,}', '1600 Pennsylvania Ave.')
<_sre.SRE_Match object; span=(0, 4), match='1600'>

>>> re.search(r'[\d]+', '1600 Pennsylvania Ave.')
<_sre.SRE_Match object; span=(0, 4), match='1600'>

The Zero Group

Up until this point, the examples have consistently used the group method to return the entire match, rather than just a single group. In fact, it may seem like very odd nomenclature indeed to have to call the group method to get back the entire match in the first place.

Why does it work this way? The purpose of the group is actually to return a single group from the match. It takes an optional argument, which is the number of the group to return. If the argument is omitted (as the examples had consistently been doing), it defaults to 0.

In regular expressions, the groups are counted based on their position in the regular expression, starting with 1.

The 0 group is special, and corresponds to the entire match. This is why groups are 1-indexed. By calling group with no argument, you are asking for group 0 and, therefore, getting the entire match back.

Named Groups

In addition to having positionally numbered groups, the Python regular expression engine also provides a mechanism for naming groups. This functionality was actually originally introduced by the Python regular expression implementation, although many other languages have picked it up at this point.

The syntax for a named group is to add ?P<group_name> immediately after the opening ( character. You could specify the local phone number regular expression to use named groups by rewriting it as (?P<first_three>[\d]{3})-(?P<last_four>[\d]{4}.

>>> match = re.search(r'(?P<first_three>[\d]{3})-(?P<last_four>[\d]{4})',
...                   '867-5309')
>>> match
<_sre.SRE_Match object; span=(0, 8), match='867-5309'>

First of all, note that named groups are also still positional groups. You can (if you choose) still look up the groups this way:

>>> match.groups()
('867', '5309')
>>> match.group(1)
'867'

Using named groups opens up two more ways to look up a group. First, the name of the group can be passed as a string to the group method.

>>> match.group('first_three')
'867'

Additionally, match objects provide a groupdict method. This method is similar in most ways to the groups method, except that it returns a dictionary instead of a tuple, and the dictionary keys correspond to the names of the groups.

>>> match.groupdict()
{'first_three': '867', 'last_four': '5309'}

It is worth noting that groupdict, like groups, does not return the entire match; it only returns the subgroups. Also, if you have a mix of named groups and unnamed groups, the unnamed groups are not part of the dictionary returned by groupdict.

>>> match = re.search(r'(?P<first_three>[\d]{3})-([\d]{4})', '867-5309')
>>> match.groups()
('867', '5309')
>>> match.groupdict()
{'first_three': '867'}

In this case, only the first group (named first_three) is a named group, and the second group is a numbered group only. Therefore, when groups is called, both groups are returned in the tuple. However, when groupdict is called, only the first_three group is included in the result.

Named groups are quite valuable for maintenance reasons. You may reference a group in code later. If you use primarily named groups, adding a new group to the regular expression to account for a change does not then require updating group numbers later in code, because the existing names stay the same.

Referencing Existing Groups

The regular expression engine also provides a mechanism to reference a previously matched group. Sometimes, you may be looking for a subsequent occurrence of the same submatch.

For example, if you are trying to parse a block of XML, you may want to very permissively look for any valid opening tag, such as <([\w_-]+)>. However, you want to ensure that the same closing tag exists.

It is insufficient to simply repeat this pattern a second time. On the one hand, it will correctly match patterns that you want.

>>> re.search(r'<([\w_-]+)>stuff</([\w_-]+)>', '<foo>stuff</foo>')
<_sre.SRE_Match object; span=(0, 16), match='<foo>stuff</foo>'>

On the other hand, it would also match patterns that should not actually match.

>>> match = re.search(r'<([\w_-]+)>stuff</([\w_-]+)>', '<foo>stuff</bar>')
>>> match
<_sre.SRE_Match object; span=(0, 16), match='<foo>stuff</bar>'>
>>> match.group(1)
'foo'
>>> match.group(2)
'bar'

Here, the regular expression engine correctly sees <foo> as an opening XML tag, matches it, and assigns the text foo to the subgroup. It then matches the literal characters stuff, and then goes to match the closing XML tag.

At this point, what you intuitively want is for the match to fail, because the closing XML tag is </bar>, which is not the same as the opening tag of <foo>.

The regular expression engine does not do that, however. It has simply been told to match the </ and > wrapping characters, and then word characters in between. Because bar fulfills this requirement, the engine matches it, assigns it to the second subgroup, and returns a match.

What you really want at this point is for the regular expression engine to require the same submatch as was used in the first group. This should make a string of <foo>stuff</foo> match, but a string of <foo>stuff</bar> fail to match.

The regular expression engine provides a way to do this using backreferences. Backreferences refer to a previously matched group within a regular expression, and cause the regular expression parser to expect the same match text to occur again.

You backreference numbered groups using \N, where N is the group number. Therefore, \1 will match the first group, \2 the second group, and so on. This syntax is capable of matching up to the first 99 groups.

Consider the following XML regular expression that uses a backreference:

>>> match = re.search(r'<([\w_-]+)>stuff</\1>', '<foo>stuff</foo>')
>>> match
<_sre.SRE_Match object; span=(0, 16), match='<foo>stuff</foo>'>
>>> match.groups()
('foo',)

Notice that there is only one subgroup now. In the previous example, there were two, both containing the text foo. In this case, however, a backreference has replaced the second group.

A much more important distinction, however, is what this regular expression does not match.

>>> re.search(r'<([\w_-]+)>stuff</\1>', '<foo>stuff</bar>')
>>>

In this case, the regular expression engine successfully matches up to the closing XML tag. However, because bar is not the same text as foo, the match fails.

Case Insensitivity

The simplest and most straightforward flag is re.IGNORECASE, which causes the regular expression to become case-insensitive.

>>> re.search(r'python', 'PYTHON IS AWESOME', re.IGNORECASE)
<_sre.SRE_Match object; span=(0, 6), match='PYTHON'>

When using re.IGNORECASE, the match will still be returned using the case of the string in which it was found, and not the case of the regular expression.

re.IGNORECASE is also aliased to re.I.

ASCII and Unicode

You may recall that there is a difference between how some character shortcuts work between Python 2 and Python 3. For example, \w in Python 3 matches word characters in nearly any language, rather than just the Latin alphabet.

The re module provides flags to make Python 2 follow the Python 3 behavior, and also flags to make Python 3 follow the Python 2 behavior.

The re.UNICODE (aliased to re.U) flag forces the regular expression engine to follow the Python 3 behavior. This flag is defined in both Python 2 and Python 3, so it is safe to use it in code designed to run on either platform. Note that if you try to use a byte string with re.U in Python 3, the parser will raise an exception.

The re.ASCII (aliased to re.A) flag forces the regular expression to follow the Python 2 behavior. Unlike re.UNICODE, the re.ASCII flag is not available in Python 2. If you need re.ASCII in code that runs under both Python 2 and Python 3, use the appropriate character classes instead, or do a version check before applying the flag.

Dot Matching Newline

The re.DOTALL flag (aliased to re.S to match the terminology used in Perl and elsewhere) causes the . character to match newline characters in addition to all other characters.

>>> re.search(r'.+', 'foo\nbar')
<_sre.SRE_Match object; span=(0, 3), match='foo'>
>>> re.search(r'.+', 'foo\nbar', re.DOTALL)
<_sre.SRE_Match object; span=(0, 7), match='foo\nbar'>

In the first command, the regular expression engine must match one or more of any character. It matches foo, and then it reaches a line break and stops, because . does not normally match line breaks.

However, in the second command, re.DOTALL is passed, and the line break character is included in what . matches against. Therefore, the regular expression engine (being greedy) keeps going until it reaches end of string, and the entire string is returned as the match.

Multiline Mode

The re.MULTILINE flag (aliased to re.M) causes the ˆ and $ characters, which normally would only match against the beginning or end of the string (respectively), to instead match against the beginning or end of any line within the string.

>>> re.search(r'ˆbar', 'foo\nbar')
>>> re.search(r'ˆbar', 'foo\nbar', re.MULTILINE)
<_sre.SRE_Match object; span=(4, 7), match='bar'>

In the first command, the ˆ character is only able to match against the beginning of the string. Therefore, the word bar does not match, because it is not the first thing in the string.

In the second command, however, the re.MULTILINE flag is used. Therefore, the ˆ character merely requires the beginning of a line. Because a newline character immediately precedes bar, it matches and the match is returned.

Verbose Mode

The re.VERBOSE flag (aliased to re.X) allows for complicated regular expressions to be expressed in a more readable way.

This flag does two things. First, it causes all whitespace (other than in character classes) to be ignored, including line breaks. Second, it treats the # character (again, unless it's inside a character class) as a comment character.

This allows for easy annotation of regular expressions, which can be valuable as they become complicated. The following two commands are equivalent:

>>> re.search(r'(?P<first_three>[\d]{3})-(?P<last_four>[\d]{4})', '867-5309')
<_sre.SRE_Match object; span=(0, 8), match='867-5309'>
>>> re.search(r"""(?P<first_three>[\d]{3})   # The first three digits
...               -                          # A literal hyphen
...               (?P<last_four>[\d]{4})     # The last four digits
...            """, '867-5309', re.VERBOSE)
<_sre.SRE_Match object; span=(0, 8), match='867-5309'>

Debug Mode

The re.DEBUG flag (not aliased) dumps some debugging information out to sys.stderr while compiling a regular expression.

>>> re.search(r'(?P<first_three>[\d]{3})-(?P<last_four>[\d]{4})',
     '867-5309', re.DEBUG)
subpattern 1
  max_repeat 3 3
    in
      category category_digit
literal 45
subpattern 2
  max_repeat 4 4
    in
      category category_digit
<_sre.SRE_Match object; span=(0, 8), match='867-5309'>

Using Multiple Flags

Occasionally, you may need to use more than one of these flags at once. To do this, join them with the | (bitwise OR) operator. For example, if you need both the re.DOTALL and re.MULTILINE flags, the correct syntax is re.DOTALL | re.MULTILINE or re.S | re.M.

Inline Flags

It is also possible to use flags within a regular expression itself by beginning the regular expression with special syntax. This uses the short-form flag, and looks like this:

>>> re.search('(?i)FOO', 'foo').group()
'foo'

Note the (?i) at the beginning. This is the equivalent of using the re.IGNORECASE flag. However, this syntax is usually less preferable to sending flags explicitly. Also, the long form of the flags will not work. (?ignorecase) is not valid and will raise an exception.